test(kubernetes): add optional Envoy-backed e2e mode by elezar · Pull Request #2259 · NVIDIA/OpenShell

elezar · 2026-07-14T12:37:21Z

Summary

Add an optional Envoy-backed execution mode for the standard single-replica Kubernetes E2E suite. The mode exercises the Gateway API/Envoy GRPCRoute path without enabling HA, changing replica counts, or requiring PostgreSQL.

Direct Service port-forwarding remains the default. Envoy mode installs or reuses an Envoy Gateway controller, creates test-owned Gateway API resources, disables Envoy request and stream duration limits for OpenShell's long-lived gRPC streams, discovers the generated proxy Service, and routes the client port-forward through that Service.

Related Issue

Extracted from the generally applicable Envoy/Gateway API setup in #1868. This PR intentionally leaves HA gateway rebalancing, multi-replica behavior, and external PostgreSQL coverage out of scope while the HA tests are not yet functional.

Changes

Add mise run e2e:kubernetes:envoy for local single-replica Envoy execution.
Add OPENSHELL_E2E_KUBE_USE_ENVOY=1 support to the Kubernetes e2e wrapper.
Add a use-envoy-gateway input to the reusable Kubernetes E2E workflow.
Add one core Kubernetes E2E matrix leg with topology: envoy and use-envoy-gateway: true.
Reuse an existing Envoy controller without upgrading or uninstalling its release.
Install Envoy with helm install only when no controller is present, using a unique test-owned namespace.
Create uniquely named test-owned GatewayClass and BackendTrafficPolicy resources, refusing to modify pre-existing resources with those names.
Delete only Envoy resources whose creation was recorded by the current invocation.
Route the client port-forward through the generated Envoy proxy Service while keeping the default direct-Service path unchanged.

Testing

bash -n e2e/with-kube-gateway.sh
git diff --check
Workflow YAML parse through the OpenShell devenv with pinned uv/PyYAML
Helm lint with ci/values-gateway.yaml
Helm render of Gateway and GRPCRoute with the per-run GatewayClass override
Focused shell simulations for existing-controller reuse, pre-existing namespace/Class refusal, and created-resource-only cleanup
mise run pre-commit completes (lint, format, workspace/E2E Clippy, Helm, Python, and other reported checks passed; the first run hit a fresh-virtualenv grpc_tools bootstrap race, and the rerun reached the five-minute command timeout during full Rust tests)
Kubernetes E2E (Docker was unavailable locally)

Checklist

Follows Conventional Commits
Commits are signed off (DCO)
Runtime mode and reusable workflow exposure are separate logical commits
Envoy coverage is single-replica and does not add HA replicas, PostgreSQL setup, or HA tests

copy-pr-bot · 2026-07-14T12:37:26Z

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

drew · 2026-07-27T15:36:48Z

+      use-envoy-gateway:
+        description: "Run the e2e command through an Envoy Gateway GRPCRoute"
+        required: false
+        type: boolean
+        default: false


What's the motivation for this? Is it to make sure we can test HA deployments?

The primary motivation for this PR (and some of the others) was to reduce the surface of #1868 (adding HA support) somewhat. This includes adding Envoy configuration / deployments to provide a stable endpoint for the HA case.

As reviewed, the PR added the option but didn't actually exercise any of the added functionality. Since using Envoy is a separate from an HA deployment, I have updated the PR to add an "envoy-enabled" path to the existing tests. This should mean that we can shake out any issues with the envoy-gateway interaction separately from the HA functionality.

Depending on the load on CI, we can consider removing it again or simplifying our test matrix for k8s once HA support lands.

Signed-off-by: Evan Lezar <elezar@nvidia.com>

elezar · 2026-07-28T08:58:00Z

/ok-to-test 9b5c53e

elezar

gator-agent

PR Review Status

Validation: This maintainer-authored PR is project-valid: it adds a scoped Envoy-backed Kubernetes E2E path split from #1868, without expanding into HA replicas or external PostgreSQL coverage. @drew asked about the motivation, and @elezar clarified that the goal is to shake out Envoy/Gateway interaction separately from HA behavior; I checked that context against the workflow and script changes.
Head SHA: 9b5c53e4ba4b7a51c95e570cd00d3ac390942653

Review findings:

Blocking: Envoy mode can expose the plaintext unauthenticated e2e gateway through a default LoadBalancer Service on existing clusters unless the test proxy is forced local/ClusterIP or gated behind an explicit unsafe opt-in.
Blocking: Envoy mode should wait for Gateway/GRPCRoute/BackendTrafficPolicy acceptance/programming before starting tests.
Warning: The new mise run e2e:kubernetes:envoy workflow and Envoy resources should be reflected in the relevant agent-maintenance/debugging docs such as .agents/skills/debug-openshell-cluster/SKILL.md and likely .agents/skills/helm-dev-environment/SKILL.md.

Docs: Fern docs are not required because this is developer/E2E workflow behavior, but agent-facing workflow docs need the companion update above.

E2E: test:e2e should run after the review findings are addressed; HA E2E is not required for this scoped Envoy path.

Next state: gator:in-review

elezar · 2026-07-29T16:08:27Z

 fi
+if use_envoy_gateway; then
+  prepare_envoy_gateway
+  helm_values_args+=(--values "${ROOT}/deploy/helm/openshell/ci/values-gateway.yaml")


gator-agent

Critical: This layers values-gateway.yaml on top of values-skaffold.yaml, which sets server.disableTls=true and server.auth.allowUnauthenticatedUsers=true. In Envoy mode on an existing cloud cluster, Envoy Gateway creates a proxy Service that defaults to LoadBalancer, so the test can expose an unauthenticated plaintext OpenShell gateway beyond the local port-forward. Please either force the test Envoy proxy Service to ClusterIP, or refuse Envoy mode for non-local contexts unless an explicit unsafe opt-in is set.

elezar · 2026-07-29T16:08:27Z

-    exit 1
-  fi
+  configure_envoy_backend_policy
+  start_gateway_portforward


gator-agent

Warning: start_gateway_portforward waits for the Envoy proxy Service and pod readiness, but not for the Gateway, GRPCRoute, or BackendTrafficPolicy to be accepted and programmed. A ready Envoy pod can still serve 404s or enforce default stream/request timeouts while reconciliation catches up. Add an Envoy-mode wait before starting tests that checks Gateway accepted/programmed, GRPCRoute parent accepted/resolved refs, and BackendTrafficPolicy acceptance when created.

danehans · 2026-07-29T21:44:56Z

xref #2469

danehans · 2026-07-29T21:51:57Z

@elezar in support of #2469, I recommend refactoring this PR to support multiple ingress implementations or adding a follow-on PR to implement a controller-neutral approach (assign me if needed):

Replace the boolean with OPENSHELL_E2E_KUBE_INGRESS=direct|envoy|agentgateway and an ingress-implementation workflow input.
Move controller behavior into small adapters such as e2e/ingress/envoy.sh and e2e/ingress/agentgateway.sh.
Give every adapter the same hooks:
- ingress_prepare
- ingress_add_helm_args
- ingress_post_install
- ingress_endpoint -> namespace, Service, port
- ingress_diagnostics
- ingress_cleanup
Keep generic Gateway/GRPCRoute condition waiting and port-forward lifecycle in with-kube-gateway.sh.
Track GATEWAY_NAME, GATEWAY_NAMESPACE and ROUTE_NAMESPACE separately from RELEASE_NAME and NAMESPACE.

The agentgateway adapter could then install or reuse its controller, create the shared Gateway fixture, configure the chart with gateway.create=false, name, namespace, and sectionName, and return agentgateway’s generated Service. Future ingress implementations would only add another adapter and CI matrix row, without further branching in the main E2E wrapper.

elezar force-pushed the codex/kubernetes-envoy-e2e branch from 0dbb332 to 8e8d0cc Compare July 27, 2026 13:26

elezar marked this pull request as ready for review July 27, 2026 14:52

elezar requested review from a team, derekwaynecarr, maxamillion and mrunalp as code owners July 27, 2026 14:52

drew reviewed Jul 27, 2026

View reviewed changes

elezar added 2 commits July 28, 2026 10:50

test(kubernetes): add safe Envoy-backed e2e mode

d5fbc67

Signed-off-by: Evan Lezar <elezar@nvidia.com>

ci(kubernetes): expose Envoy Gateway e2e input

9b5c53e

Signed-off-by: Evan Lezar <elezar@nvidia.com>

elezar force-pushed the codex/kubernetes-envoy-e2e branch from 8e8d0cc to 9b5c53e Compare July 28, 2026 08:51

elezar added the gator:in-review Gator is reviewing or awaiting PR review feedback label Jul 29, 2026

elezar commented Jul 29, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

test(kubernetes): add optional Envoy-backed e2e mode - #2259

test(kubernetes): add optional Envoy-backed e2e mode#2259
elezar wants to merge 2 commits into
mainfrom
codex/kubernetes-envoy-e2e

elezar commented Jul 14, 2026 •

edited

Loading

Uh oh!

copy-pr-bot Bot commented Jul 14, 2026

Uh oh!

drew Jul 27, 2026

Uh oh!

elezar Jul 28, 2026 •

edited

Loading

Uh oh!

elezar commented Jul 28, 2026

Uh oh!

elezar left a comment

Uh oh!

elezar Jul 29, 2026

Uh oh!

elezar Jul 29, 2026

Uh oh!

danehans commented Jul 29, 2026 •

edited

Loading

Uh oh!

danehans commented Jul 29, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

elezar commented Jul 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Related Issue

Changes

Testing

Checklist

Uh oh!

copy-pr-bot Bot commented Jul 14, 2026

Uh oh!

drew Jul 27, 2026

Choose a reason for hiding this comment

Uh oh!

elezar Jul 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

elezar commented Jul 28, 2026

Uh oh!

elezar left a comment

Choose a reason for hiding this comment

PR Review Status

Uh oh!

elezar Jul 29, 2026

Choose a reason for hiding this comment

Uh oh!

elezar Jul 29, 2026

Choose a reason for hiding this comment

Uh oh!

danehans commented Jul 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

danehans commented Jul 29, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

elezar commented Jul 14, 2026 •

edited

Loading

elezar Jul 28, 2026 •

edited

Loading

danehans commented Jul 29, 2026 •

edited

Loading